<html>
<head><meta charset="utf-8"><title>Parallelism in the compiler · t-compiler/help · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/index.html">t-compiler/help</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html">Parallelism in the compiler</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="233037466"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233037466" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Timothy Maloney <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233037466">(Apr 03 2021 at 22:24)</a>:</h4>
<p>I read that one of the current issues in the rust compiler is that the only time parallelism is used is in codegen.</p>
<p>I am curious about possibly implementing the lexer to include parallelism, <br>
but assume someone else is maybe working on this, or things like this have already been considered.</p>
<p>Would someone be able to point me in the right direction?</p>



<a name="233037872"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233037872" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> cjgillot <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233037872">(Apr 03 2021 at 22:31)</a>:</h4>
<p>The compiler has some support for parallelism before codegen when compiled in the <code>parallel-compiler</code> configuration. This configuration is not well tested, so it's disable by default. I don't know if the lexer parallelism is implemented yet.</p>



<a name="233039518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233039518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233039518">(Apr 03 2021 at 23:03)</a>:</h4>
<p>I think parallelizing lexing is unlikely to have much of an impact because it takes little time compared to the rest of the compiler. Extending the query system to include parts of the compiler frontend and then using the query-level parallelization that's already there seems like a more promising approach.</p>



<a name="233040067"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233040067" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Timothy Maloney <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233040067">(Apr 03 2021 at 23:15)</a>:</h4>
<p>When you say compiler front end, do you mean the lexer? If that's the case I'm assuming that would then mean extending the query system to be used in the lexing/parsing of the source? Just for clarification.</p>



<a name="233040285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233040285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233040285">(Apr 03 2021 at 23:20)</a>:</h4>
<p>Yes, and also within name resolution and macro expansion.</p>



<a name="233040299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233040299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233040299">(Apr 03 2021 at 23:20)</a>:</h4>
<p>I don't think lexing should itself be a query, but parsing a file or macro expansion might</p>



<a name="233040425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233040425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233040425">(Apr 03 2021 at 23:23)</a>:</h4>
<p><span class="user-mention silent" data-user-id="211727">Jonas Schievink  [he/him]</span> <a href="#narrow/stream/182449-t-compiler.2Fhelp/topic/Parallelism.20in.20the.20compiler/near/233039518">said</a>:</p>
<blockquote>
<p>I think parallelizing lexing is unlikely to have much of an impact because it takes little time compared to the rest of the compiler. Extending the query system to include parts of the compiler frontend and then using the query-level parallelization that's already there seems like a more promising approach.</p>
</blockquote>
<p>some recent discussion of this: <a href="#narrow/stream/241847-t-compiler.2Fwg-incr-comp/topic/Design.20of.20HIR">https://rust-lang.zulipchat.com/#narrow/stream/241847-t-compiler.2Fwg-incr-comp/topic/Design.20of.20HIR</a></p>



<a name="233040531"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233040531" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233040531">(Apr 03 2021 at 23:25)</a>:</h4>
<p>To clarify, this isn't necessarily a blocker for having parallel queries, it just makes them more effective. If your goal is simply to add more parallelism to rustc, then looking into the existing parallel query engine and improving it (for example, to reduce the overhead compared to a single-threaded compiler) is probably the way to go.</p>



<a name="233044985"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233044985" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233044985">(Apr 04 2021 at 00:55)</a>:</h4>
<p><code>parallel-compiler = true</code> doesn't help with building the stage1 compiler either since that just uses the beta one, right?</p>



<a name="233045097"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233045097" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233045097">(Apr 04 2021 at 00:57)</a>:</h4>
<p>I am not sure what you mean by the stage 1 compiler (the wording is ambiguous: <a href="https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html#understanding-stages-of-bootstrap">https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html#understanding-stages-of-bootstrap</a>). The bootstrap compiler doesn't have parallel-compiler = true set, no</p>



<a name="233045728"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233045728" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233045728">(Apr 04 2021 at 01:08)</a>:</h4>
<p>e.g. when running <code>./x.py test src/test/ui --stage 1</code>, that builds a rustc (which takes several minutes) and that won't benefit from <code>parallel-compiler = true</code></p>



<a name="233045751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233045751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233045751">(Apr 04 2021 at 01:09)</a>:</h4>
<p>correct</p>



<a name="233046107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233046107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jonas Schievink  [he/him] <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233046107">(Apr 04 2021 at 01:17)</a>:</h4>
<p>the resulting compiler will though</p>



<a name="233046168"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233046168" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233046168">(Apr 04 2021 at 01:18)</a>:</h4>
<p>that's not useful for testing because they're already executed in parallel</p>



<a name="233046188"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/182449-t-compiler/help/topic/Parallelism%20in%20the%20compiler/near/233046188" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/182449-t-compiler/help/topic/Parallelism.20in.20the.20compiler.html#233046188">(Apr 04 2021 at 01:18)</a>:</h4>
<p>well, depending what you mean by benefit - I think some UI tests are non-deterministic with multiple threads <span aria-label="upside down" class="emoji emoji-1f643" role="img" title="upside down">:upside_down:</span> <a href="https://github.com/rust-lang/rust/issues/75760">https://github.com/rust-lang/rust/issues/75760</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>